Reliability, validity and acceptability of an online clinical reasoning simulator for medical students: An international pilot.

INTRODUCTION: Clinical reasoning skills are essential for decision-making. Current assessment methods are limited when testing clinical reasoning and management of uncertainty. This study evaluates the reliability, validity and acceptability of Practicum Script, an online simulation-based programme, for developing medical students' clinical reasoning skills using real-life cases. METHODS: In 2020, we conducted an international, multicentre pilot study using 20 clinical cases with 2457 final-year medical students from 21 schools worldwide. Psychometric analysis was performed (n = 1502 students completing at least 80% of cases). Classical estimates of reliability for three test domains (hypothesis generation, hypothesis argumentation and knowledge application) were calculated using Cronbach's alpha and McDonald's omega coefficients. Validity evidence was obtained by confirmatory factor analysis (CFA) and measurement alignment (MA). Items from the knowledge application domain were analysed using cognitive diagnostic modelling (CDM). Acceptability was evaluated by an anonymous student survey. RESULTS: Reliability estimates were high with narrow confidence intervals. CFA revealed acceptable goodness-of-fit indices for the proposed three-factor model. CDM analysis demonstrated good absolute test fit and high classification accuracy estimates. Student survey responses showed high levels of acceptability. CONCLUSION: Our findings suggest that Practicum Script is a useful resource for strengthening students' clinical reasoning skills and ability to manage uncertainty.

UK Prescribing Safety Assessment (PSA): The development, implementation and outcomes of a national online prescribing assessment.

AIMS: The United Kingdom (UK) Prescribing Safety Assessment (PSA) is a 2-h online assessment of basic competence to prescribe and supervise the use of medicines. It has been undertaken by students and doctors in UK medical and foundation schools for the past decade. This study describes the academic characteristics and performance of the assessment; longitudinal performance of candidates and schools; stakeholder feedback; and surrogate markers of prescribing safety in UK healthcare practice. METHODS: We reviewed the performance data generated by over 70 000 medical students and 3700 foundation doctors who have participated in the PSA since its inception in 2013. These data were supplemented by Likert scale and free text feedback from candidates and a variety of stakeholder groups. Further data on medication incidents, collected by national reporting systems and the regulatory body, are reported, with permission. RESULTS: We demonstrate the feasibility, high quality and reliability of an online prescribing assessment, uniquely providing a measure of prescribing competence against a national standard. Over 90% of candidates pass the PSA on their first attempt, while a minority are identified for further training and assessment. The pass rate shows some variation between different institutions and between undergraduate and foundation cohorts. Most responders to a national survey agreed that the PSA is a useful instrument for assessing prescribing competence, and an independent review has recommended adding the PSA to the Medical Licensing Assessment. Surrogate markers suggest there has been improvement in prescribing safety in practice, temporally associated with the introduction of the PSA but other factors could be influential too. CONCLUSIONS: The PSA is a practical and cost-effective way of delivering a reliable national assessment of prescribing competence that has educational impact and is supported by the majority of stakeholders. There is a need to develop national systems to identify and report prescribing errors and the harm they cause, enabling the impact of educational interventions to be measured.

Is the awarding gap at UK medical schools influenced by ethnicity and medical school attended? A retrospective cohort study.

OBJECTIVES: To better characterise the Awarding Gap (AG) between black, Asian and other minority ethnic (BAME) and white students in UK undergraduate medical education by examining how it affects eight minority ethnicity subgroups (Bangladeshi, black, Chinese, Indian, Pakistani, mixed, other Asian background and other ethnic background) and whether the AG varies by medical school attended. DESIGN: Retrospective cohort study. SETTING: Data extracted from the UK Medical Education Database on students enrolled at 33 UK medical schools in the academic years starting 2012, 2013 and 2014. PARTICIPANTS: 16 020 'Home' tuition fee status students who sat the University Clinical Aptitude Test on entry to university and obtained a UK Foundation Programme (UKFP) application score on exit. PRIMARY OUTCOME MEASURE: UKFP Z-scores on exit from medical school. RESULTS: There were significant differences in UKFP Z-scores between ethnicity subgroups. After white students, mixed ethnicity students performed best (coefficient -0.15 standard deviations [SD]) compared with white students, (95% confidence interval [CI] -0.23 to -0.08, p<0.001) and Pakistani students scored lowest (coefficient -0.53 SD, 95% CI -0.60 to -0.46, p<0.001). In pairwise comparisons of scores between all nine individual ethnicity subgroups, 15/36 were statistically significant. The AG varied considerably across medical schools. The largest gap showed the coefficient for BAME was -0.83 SD compared with white students (95% CI -1.18 to -0.49, p<0.001), while the smallest demonstrated no statistically significant difference in performance between BAME and white students (+0.05 SD, 95% CI -0.32 to 0.42, p=0.792). CONCLUSIONS: BAME students are significantly disadvantaged by the current UK medical education system. There are clear differences in medical school outcomes between students from different ethnicity subgroups, and the size of the AG also varies by medical school attended. Urgent and effective action must be taken to address the AG and achieve an equal learning environment for our future doctors.

The validity of Engagement and Feedback Assessments (EFAs): identifying students at risk of failing.

BACKGROUND: Imperial College School of Medicine, London UK, introduced a new curriculum in 2019, with a focus on the GMC outcomes for graduates, and pedagogy best practice. The new curriculum included formative assessments, named engagement and feedback assessments (EFAs), to support learning, and attainment in the summative examinations. The aims of this study were to assess the validity of EFAs and to determine whether they have utility as a modified form of programmatic assessment to inform decision-making regarding possible interventions by measuring and analysing attendance at and performance in these formative events. METHODS: Seven hundred and sixty-one students were included in the study and assessment results were included for academic years 2019/20 to 2020/21. Forty-one data points per student, (27 in Year 1 and 14 in Year 2) were used, to compare EFA scores with the summative performance. Attendance was monitored through engagement with the EFAs. RESULTS: Cohort 1 (enrolled 2019): In year 1, EFAs were associated with summative exam scores (overall r = 0.63, p < 0.001). Year 2, EFA scores were also associated with summative scores (overall r = 0.57, p < 0.001), including the clinical practical assessment (r = 0.45, p < 0.001). Missing two or more EFAs was associated with a significant increase in the likelihood of failing one or more summative examinations in the first year (OR: 7.97, 95% CI 2.65-34.39) and second year (OR: 3.20, 95% CI 1.74-5.95). Missing more than two EFAs in their first year was also associated with a higher risk of failing a summative examination in the second year (OR: 2.47, 95% CI 1.33-4.71). Students who increased their attendance between year 1 and 2 fared better in summative assessment than those who maintained poor attendance, whereas those that reduced their attendance fared worse than those that maintained high attendance. Cohort 2 (enrolled 2020): Analysis of cohort 2 supported these findings and in this cohort missing two or more EFAs was again associated with an increased likelihood of failing a summative examination (OR = 4.00, 95% CI = 2.02-7.90). CONCLUSION: Our EFA model has validity in predicting performance in summative assessments and can inform prospective interventions to support students' learning. Enhancing attendance and engagement can improve outcomes.

Natural history of non-functioning pituitary microadenomas: results from the UK non-functioning pituitary adenoma consortium.

OBJECTIVE: The optimal approach to the surveillance of non-functioning pituitary microadenomas (micro-NFPAs) is not clearly established. Our aim was to generate evidence on the natural history of micro-NFPAs to support patient care. DESIGN: Multi-centre, retrospective, cohort study involving 23 endocrine departments (UK NFPA consortium). METHODS: Clinical, imaging, and hormonal data of micro-NFPA cases between January, 1, 2008 and December, 21, 2021 were analysed. RESULTS: Data for 459 patients were retrieved [median age at detection 44 years (IQR 31-57)-152 males/307 females]. Four hundred and nineteen patients had more than two magnetic resonance imagings (MRIs) [median imaging monitoring 3.5 years (IQR 1.71-6.1)]. One case developed apoplexy. Cumulative probability of micro-NFPA growth was 7.8% (95% CI, 4.9%-8.1%) and 14.5% (95% CI, 10.2%-18.8%) at 3 and 5 years, respectively, and of reduction 14.1% (95% CI, 10.4%-17.8%) and 21.3% (95% CI, 16.4%-26.2%) at 3 and 5 years, respectively. Median tumour enlargement was 2 mm (IQR 1-3) and 49% of micro-NFPAs that grew became macroadenomas (nearly all >5 mm at detection). Eight (1.9%) patients received surgery (only one had visual compromise with surgery required >3 years after micro-NFPA detection). Sex, age, and size at baseline were not predictors of enlargement/reduction. At the time of detection, 7.2%, 1.7%, and 1.5% patients had secondary hypogonadism, hypothyroidism, and hypoadrenalism, respectively. Two (0.6%) developed hypopituitarism during follow-up (after progression to macroadenoma). CONCLUSIONS: Probability of micro-NFPA growth is low, and the development of new hypopituitarism is rare. Delaying the first follow-up MRI to 3 years and avoiding hormonal re-evaluation in the absence of tumour growth or clinical manifestations is a safe approach for micro-NFPA surveillance.

Knowledge Attainment and Engagement Among Medical Students: A Comparison of Three Forms of Online Learning.

Objective: This study compared knowledge attainment and student enjoyment and engagement between clinical case vignette, patient-testimony videos and mixed reality (MR) teaching via the Microsoft HoloLens 2, all delivered remotely to third year medical students. The feasibility of conducting MR teaching on a large scale was also assessed. Setting & Participants: Medical students in Year 3 at Imperial College London participated in three online teaching sessions, one in each format. All students were expected to attend these scheduled teaching sessions and to complete the formative assessment. Inclusion of their data used as part of the research trial was optional. Primary and Secondary Outcome Measures: The primary outcome measure was performance on a formative assessment, which served to compare knowledge attainment between three forms of online learning. Moreover, we aimed to explore student engagement with each form of learning via a questionnaire, and also feasibility of applying MR as a teaching tool on a large scale. Comparisons between performances on the formative assessment between the three groups were investigated using a repeated measures two-way ANOVA. Engagement and enjoyment were also analysed in the same manner. Results: A total of 252 students participated in the study. Knowledge attainment of students using MR was comparable with the other two methods. Participants reported higher enjoyment and engagement (p<0.001) for the case vignette method, compared with MR and video-based teaching. There was no difference in enjoyment or engagement ratings between MR and the video-based methods. Conclusion: This study demonstrated that the implementation of MR is an effective, acceptable, and feasible way of teaching clinical medicine to undergraduate students on a large scale. However, case-based tutorials were found to be favoured most by students. Future work could further explore the best uses for MR teaching within the medical curriculum.

The impact of patient skin colour on diagnostic ability and confidence of medical students.

Previous literature has explored unconscious racial biases in clinical education and medicine, finding that people with darker skin tones can be underrepresented in learning resources and managed differently in a clinical setting. This study aimed to examine whether patient skin colour can affect the diagnostic ability and confidence of medical students, and their cognitive reasoning processes. We presented students with 12 different clinical presentations on both white skin (WS) and non-white skin (NWS). A think aloud (TA) study was conducted to explore students' cognitive reasoning processes (n = 8). An online quiz was also conducted where students submitted a diagnosis and confidence level for each clinical presentation (n = 185). In the TA interviews, students used similar levels of information gathering and analytical reasoning for each skin type but appeared to display increased uncertainty and reduced non-analytical reasoning methods for the NWS images compared to the WS images. In the online quiz, students were significantly more likely to accurately diagnose five of the 12 clinical presentations (shingles, cellulitis, Lyme disease, eczema and meningococcal disease) on WS compared to NWS (p < 0.01). With regards to students' confidence, they were significantly more confident diagnosing eight of the 12 clinical presentations (shingles, cellulitis, Lyme disease, eczema, meningococcal disease, urticaria, chickenpox and Kawasaki disease) on WS when compared to NWS (p < 0.01). These findings highlight the need to improve teaching resources to include a greater diversity of skin colours exhibiting clinical signs, to improve students' knowledge and confidence, and ultimately, to avoid patients being misdiagnosed due to the colour of their skin.

Validity of constructed-response situational judgment tests in training programs for the health professions: A systematic review and meta-analysis protocol.

BACKGROUND: Situational judgments tests have been increasingly used to help training programs for the health professions incorporate professionalism attributes into their admissions process. While such tests have strong psychometric properties for testing professional attributes and are feasible to implement in high-volume, high-stakes selection, little is known about constructed-response situational judgment tests and their validity. METHODS: We will conduct a systematic review of primary published or unpublished studies reporting on the association between scores on constructed-response situational judgment tests and scores on other tests that measure personal, interpersonal, or professional attributes in training programs for the health professions. In addition to searching electronic databases, we will contact academics and researchers and undertake backward and forward searching. Two reviewers will independently screen the papers and decide on their inclusion, first based on the titles and abstracts of all citations, and then according to the full texts. Data extraction will be done independently by two reviewers using a data extraction form to chart study details and key findings. Studies will be assessed for the risk of bias and quality by two reviewers using the "Quality In Prognosis Studies" tool. To synthesize evidence, we will test the statistical heterogeneity and conduct a psychometric meta-analysis using a random-effects model. If adequate data are available, we will explore whether the meta-analytic correlation varies across different subgroups (e.g., race, gender). DISCUSSION: The findings of this study will inform best practices for admission and selection of applicants for training programs for the health professions and encourage further research on constructed-response situational judgment tests, in particular their validity. TRIAL REGISTRATION: The protocol for this systematic review has been registered in PROSPERO [CRD42022314561]. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022314561.

Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculum.

Most undergraduate written examinations use multiple-choice questions, such as single best answer questions (SBAQs) to assess medical knowledge. In recent years, a strong evidence base has emerged for the use of very short answer questions (VSAQs). VSAQs have been shown to be an acceptable, reliable, discriminatory, and cost-effective assessment tool in both formative and summative undergraduate assessments. VSAQs address many of the concerns raised by educators using SBAQs including inauthentic clinical scenarios, cueing and test-taking behaviours by students, as well as the limited feedback SBAQs provide for both students and teachers. The widespread use of VSAQs in medical assessment has yet to be adopted, possibly due to lack of familiarity and experience with this assessment method. The following twelve tips have been constructed using our own practical experience of VSAQs alongside supporting evidence from the literature to help medical educators successfully plan, construct and implement VSAQs within medical curricula.

Effectiveness of short, personalised student assistantships: an evaluative study across eight London hospitals.

OBJECTIVES: Student assistantships are recommended to prepare medical graduates for clinical practice. Traditionally, assistantships have consisted of longer placements, often up to 15 weeks. However, within the constraints of the final year, medical schools need to carefully balance the time required for specialty placements, assessments and the risk of student burnout. We set out to evaluate the effectiveness of shorter, personalised student assistantships. DESIGN: An evaluative study on the changes in final year student confidence in preparedness for practice after a 3-week assistantship with defined learning objectives and learning needs assessment. SETTING: Eight hospitals affiliated with Imperial College School of Medicine. OUTCOMES: Student confidence in 10 learning outcomes including organising ward rounds, documentation, communication with colleagues, communication with patients and relatives, patient handover, practical procedures, patient management, acute care, prioritisation and out-of-hours clinical work. RESULTS: Two hundred and twenty final year medical students took part in the student assistantship, of whom 208 completed both the pre-assistantship and post-assistantship confidence rating questionnaires (95% completion rate). After the assistantship, 169 (81%) students expressed increased confidence levels in one or more learning objectives. For each individual learning objective, there was a significant change in the proportion of students who agreed or strongly agreed after the assistantship (p<0.0001). CONCLUSION: Overall, the focused 3-week, personalised student assistantships led to significant improvement across all learning objectives related to preparedness for practice. The use of the pre-assistantship confidence rating questionnaire allowed students to identify and target areas of learning needs during their assistantship.

Effects of Elaboration and Instructor Feedback on Retention of Clinical Reasoning Competence Among Undergraduate Medical Students: A Randomized Crossover Trial.

This randomized crossover trial examines whether elaboration on common errors in patient treatment, combined with individualized mailed feedback, improves medium-term retention of clinical reasoning competence.

Time for preference-informed foundation allocation?

Successful completion of year 1 of the UK Foundation Programme is a General Medical Council requirement that newly qualified doctors must achieve in order to gain full registration for licence to practise in the UK. We present compelling evidence that both sections of the UK Foundation Programme allocation process, consisting of the Educational Performance Measure and Situational Judgement Test scores, are not fit for purpose. The ranking process drives competitive behaviours among medical students and undermines NHS teamworking values. Furthermore, data from 2013-2020 show that UK minority ethnic students consistently receive significantly lower SJT scores than White students. The current process in the UK allocates lower ranked students, who often need more academic and social support, to undersubscribed regions. This can lead to vacancies in less popular regions, ultimately worsening health inequality. A preference-informed allocation process will improve trainee access to support and help retain trainees in underserved regions. We aim to summarise the flaws of the current system and report a potential radical solution.

Is the assumption of equal distances between global assessment categories used in borderline regression valid?

BACKGROUND: Standard setting for clinical examinations typically uses the borderline regression method to set the pass mark. An assumption made in using this method is that there are equal intervals between global ratings (GR) (e.g. Fail, Borderline Pass, Clear Pass, Good and Excellent). However, this assumption has never been tested in the medical literature to the best of our knowledge. We examine if the assumption of equal intervals between GR is met, and the potential implications for student outcomes. METHODS: Clinical finals examiners were recruited across two institutions to place the typical 'Borderline Pass', 'Clear Pass' and 'Good' candidate on a continuous slider scale between a typical 'Fail' candidate at point 0 and a typical 'Excellent' candidate at point 1. Results were analysed using one-sample t-testing of each interval to an equal interval size of 0.25. Secondary data analysis was performed on summative assessment scores for 94 clinical stations and 1191 medical student examination outcomes in the final 2 years of study at a single centre. RESULTS: On a scale from 0.00 (Fail) to 1.00 (Excellent), mean examiner GRs for 'Borderline Pass', 'Clear Pass' and 'Good' were 0.33, 0.55 and 0.77 respectively. All of the four intervals between GRs (Fail-Borderline Pass, Borderline Pass-Clear Pass, Clear Pass-Good, Good-Excellent) were statistically significantly different to the expected value of 0.25 (all p-values < 0.0125). An ordinal linear regression using mean examiner GRs was performed for each of the 94 stations, to determine pass marks out of 24. This increased pass marks for all 94 stations compared with the original GR locations (mean increase 0.21), and caused one additional fail by overall exam pass mark (out of 1191 students) and 92 additional station fails (out of 11,346 stations). CONCLUSIONS: Although the current assumption of equal intervals between GRs across the performance spectrum is not met, and an adjusted regression equation causes an increase in station pass marks, the effect on overall exam pass/fail outcomes is modest.

Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having access to the answers make a difference?

BACKGROUND: We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations. METHODS: Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set. RESULTS: A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z = -2.82, p = 0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z = 2.46, p = 0.014). CONCLUSIONS: When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.

BMAT's predictive validity for medical school performance: A retrospective cohort study.

BACKGROUND: Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools. METHODS: This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students. RESULTS: When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades. DISCUSSION: Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.